The invention concerns a method for accurately determining spatial and/or time intervals in the focused image of a lens/aperture-diaphragm system, herein called lens-diaphragm system, and/or of spatial and/or time parameters such as speed or depth of an object, in particular to accurately and intelligently control robots.
Cameras operating in a geometric-optics manner are already used in passive methods and they image onto photographic films or on arrays of charge-coupled devices (CCD). Both procedures provide spatial 3-D information in flat, 2-D form in the image plane. While the resolution of optic cameras is restricted only by optical parameters and by film grain size, on the other hand they are limited to storage and are unsuitable for processing in real time. While CCD cameras offer rapid image processing, they are, however, limited in their resolution by the pixel size of the CCD arrays. It is true that holographically imaging and recording devices can both provide and store spatial 3-D information, but only in coherent light. Moreover, a 2-beam procedure, namely an object and a reference beam, are inherently required in picture-taking. Lastly, holographic systems are able to process images in real time only under constrained conditions. Stereo cameras are technically even more complex and demand computations based on two CCD arrays to obtain 3D information.
As regards the technology of optic sensors, and in particular optical-grating spatial-frequency-filtering using the CORREVIT-/CORREFOT measurement technique (CORREVIT is a trademark of CORRSYS GmbH, Wetzlar, Germany), signal processing regarding length, speed and interval measurements of objects moving relative to the sensor (assembly line, industrial conveyor belts and such) are implemented by inserting gratings between geometric-optics imaging means and following photodiodes into the optic beam to act as signal modulators, as described for instance in German Patent Documents A 21 44 487 and AS 2156617. However, the gratings are used therein merely as 1D-/2D- beam splitters, not as diffraction 1D-/2D-/3D- gratings.
Theoretical modeling was carried out at the "Institut fur Optosensorik Dr. Lauinger/Wetzlar" based on a 3D grating and relating to the "inverted retina of the human eye" that were considered relating (N. Lauinger, "A New Interpretation Of The Stiles-Crawford Effects In Human Vision", Journal of Biological Physics, 19: 167-188, 1994; N. Lauinger, "The Relationship Between Brightness, Hue And Saturation When The Inverted Human Retina Is Interpreted As A Cellular Diffractive 3D Chip", SPIE Proceedings, Vol. 2588, Oct. 1995, 208-232) to subjective phenomena known in human vision (aperture effects, the so-called Stiles-Crawford effects I and II, the Bezold bridge phenomenon, for example). Partial aspects were computed by modeling in 4D grating optics (M. Carbon, "Using Diffraction Theory Of Human Vision For Design Of Color Vision Devices", SPIE Proceedings, Vol. 2353, 1994, 550-560; N. Lauinger, "Inverted Retina Of The Human Eye; A Trichromatic 4D Space-Time Optical Correlator", SPIE Proceedings, Vol. 2904, 1996, 344-360). Effects of a diffraction 3D grating were described regarding incoherence-coherence transformation of optic radiation, as known from German Patent Document A 37 40 533.
In spite of vast and comprehensive endeavors especially regarding automation and robotics, there is to date no technically high resolution, rugged and real time determination of 3D motion based on contactless sensor data acquisition. The core significance of optical flux fields for visual determination of motion is known from German Patent Document A 195 03 606, also "Structure From Motion By Linear Transformation", Proc. Workshop "Kognitive Robotiki", B. Krieg-Bruckner, Ch. Herwig. Edts., ZKW-Bericht 3/95, Zentrum fur Kognitionswissenschaften Universitat Bremen, March 1995. The essential performance features of neural networks are parallelism and adaptiveness (H. O. Carmesin, "Theorie Neuronaler Adaption", Koster, Berlin 1994; H. O. Carmesin, "Neurophysics Of Adaption", Physics Essays, 8(1), 38-51, 1995). Even though neural networks are especially well suited for self-adjusting, high resolution image processing, the neural adaptation theory so far has been used neither in determining motion nor in analyzing diffraction reflex sequences in imaging sensors.
The same problems also arise outside the visible spectrum.
The purpose of the invention is to provide a method and apparatus of the initially cited kind allowing accurate determination in practically real time of spatial and/or time intervals in focused image sequences of a lens-diaphragm system and/or of spatial and/or time-related object-parameters such as speed or depth.
The invention is based on the surprising insight that electromagnetic beams focusable in the image plane with a length resolution of about the light wavelength can be modulated with an accuracy of a wavelength in a position-specific manner by a 3D modulator. The electromagnetic beams so modulated subsequently can be detected in a detector of lesser length resolution and the positions in the image plane can be calculated using the position-specific modulation. As a result, the 3D structure of the light field, and hence the 3D structure of objects, can be detected with high accuracy. Illustratively, electromagnetic beams diffracted at a grating assume a direction determined by the crystal orientation corresponding to the condition of reflection and a wavelength corresponding to the Bragg condition.
The advantages offered by the invention are illustratively discussed below for the visible spectrum:
(1) At present the accuracy of determining the intervals, i.e., spacings of a brightness distribution in a focal plane, is bound by the pixel size of an imaging sensor, these pixels being about 20 micrometer (Am). Using a special embodiment of the apparatus of the present invention, that is with a 3D modulator in the form of a 3D diffraction grating, this accuracy will be limited by the wavelength of the light and by the grating spacing. When selecting an optimal grating, such values are about 0.4 .mu.m. Therefore, the length resolution can be increased by a factor of 50 and the surface resolution by a factor of 2,500.
Such an increase in accuracy is important because the human eye comprises in its retina more than 10.sup.8 receptors each with a corresponding length resolution of about 1 .mu.m, because the correspondingly generated cultural environments in the household, at leisure, in traffic and at work are designed for visual systems such as man's, and because future intelligent mobile automatons and robots require comparable visual systems so they may be operated reliably and economically.
(2) The method of the invention allows analyzing by correlation optics and in a simple and economic manner spatial and/or time-related object-parameters such as speeds or depths.
(3) The method of the invention allows universally using algorithms in the form of learning neural networks at appropriately intended sites for particular applications and for corresponding, specific apparatus, without having to develop a new algorithm. Thereby typically occurring development costs are precluded from the beginning.
(4) In the limiting case of large distances from the grating (far field), the theory of diffraction is already very well developed. It is widely used in optics and materials research. On the other hand, the physics of the near field is comparatively complex and is application-specific. Accordingly, the theory might be developed further and to-date no applications are known relating to the determination of motion, while on the other hand the learning, neural networks used herein are applicable on account of flexibility and are especially adequate. Generally, the central significance of flow fields in determining motion has been known since Helmholtz and, moreover, beyond distance of five meters, determining motion from flow fields is advantageous over that from using binoculars, and quite generally visual determination of spatial and/or time-related object-parameters offers the advantage of high resolution, large range, high speed and natural radiation sources when compared to other modes of operation.
(5) A light field in general contains both amplitude and phase information. In most technical visual systems, only amplitude information is used. On the other hand, in the method of the invention and using the 3D diffraction grating, use also may be made of the latent 3D phase information, even when using incoherent light as emitted by natural light sources. Presumably, in natural visual systems, advantageous analysis of phase information in the human eye takes place by means of the 3D diffraction grating of the inverted retina.
(6) In recent decades, explosive development has taken place in neural networks. The essential advantages are parallelism and adaptivity. Thereby, the neural networks are especially well suited to self-adjusting and high resolution image processing in real time even in complex analyses. In a special implementation of the method of the invention, the neural networks may be used at a site where they can in fact perform their best, in the parallel high resolution reconstruction of object-parameters from comprehensive, as modulated as possible, and displaced diffraction patterns.
Further purposes, advantages, features and applications of the invention are elucidated in the following description of illustrative embodiments of the invention and in relation to the attached drawings. All described and/or graphically shown features per se or in arbitrary meaningful combination represent the objective of the invention, even independently of their summary versions in the claims or of relationships between the claims.
FIG. 1 shows a geometrically/optically imaging lens-diaphragm system with a 3D grating in the image plane,
FIG. 2 is a schematic of a possible embodiment of apparatus of the invention,
FIG. 3 shows a concrete embodiment of the apparatus of the invention,
FIG. 4 schematically shows the imaging and resolving configuration of a concrete embodiment of the apparatus of the invention,
FIG. 5a shows a further embodiment of the apparatus of the invention,
FIG. 5b is a detail of FIG. 5a,
FIG. 6 is an illustrative embodiment of a pattern Fourier-Transform,
FIG. 7 shows the Fourier-Transform using a volume holographic optic element,
FIG. 8 shows the Fourier Transform of a bar structure generated by means of the volume holographic element of FIGS. 7 and 8,
FIG. 9 shows the configuration when using a curved 3D diffraction grating of a further embodiment of the apparatus of the invention,
FIG. 10 schematically shows the state when implementing a learning phase of one implementing mode of the method of the invention,
FIG. 11 shows an illustrative embodiment to determine a depth card using the 3D Talbot effect,
FIG. 12 shows the configuration of the 3D diffraction grating with detectors relating to the illustration of FIG. 11,
FIGS. 13, 14 show the intensity distributions at the Bragg grating in front of the focal point,
FIG. 15 shows the superposition of the intensity distributions at the Bragg grating behind the focal point,
FIG. 16 shows the design of the 3D diffraction grating with detector elements in relation to the configuration of FIG. 11, however when using white light and the luminous element being off-axis,
FIGS. 17, 18, show the intensity distribution of the Bragg grating before the focus of FIG. 16,
FIGS. 19, 20 show the intensity distributions for the Bragg grating behind the focus, and
FIG. 21 shows a configuration relating to the 3D Talbot effect.